Call the baseline a validation baseline - #468
Merged
Merged
Conversation
scripts/catalogue_validating.txt was described as 'the supported list'. It records one thing: which examples pass 'orchestrator validate'. That is not the same as running, producing correct artifacts, or being a supported example, and one name for four different claims is how a weak signal gets read as a strong one. Renamed to catalogue_validation_baseline.txt, with the distinction stated in the file itself, the script docstring and the CI job comment. examples/ supported/ remains the stronger contract: those are executed and their behaviour asserted. No behaviour change. The gate still fails on a named file leaving the baseline and still ignores the headline count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the review's point that the catalogue dashboard is not yet a product-support dashboard.
The problem with the old name
scripts/catalogue_validating.txtwas described in the script as "the supported list". It records exactly one thing: which examples passorchestrator validate.That is not the same as:
One name carrying four claims is how a weak signal comes to be read as a strong one — and this signal is genuinely weak. 27 of the current failures are
Undefined variable, which says nothing about whether the other 50 do anything useful.The change
Renamed to
scripts/catalogue_validation_baseline.txt(viagit mv, so history follows), with the distinction stated in three places people actually read:examples/supported/remains the stronger contract: those are executed and their behaviour asserted. This file is the weaker, wider net, and now says so.No behaviour change
The gate still fails when a named file leaves the baseline and still ignores the headline count. 831 passed, catalogue 50/117, all 50 baseline files still validate.
Still outstanding from the same review point
Moving the dashboard from scraped first-error text to stable error codes is separate, and is now genuinely possible: #467 exposes
ValidationIssue.codethroughvalidate --json, so the report can consume structured findings instead of grouping human-readable strings. The semantic classification (supported / live-dependent / legacy / duplicate / archive) remains a judgement that lives in issue #104.🤖 Generated with Claude Code